Conversation-and-control: extending speech controlled graphical user interfaces

Todays standard input devices for graphical user interfaces, mouse and keyboard, cannot be utilized by users whose motor functions are limited. Examples include users suffering from motor restrictions due to illnesses and users in application domains with manual activities. This dissertation engages in the research area speech-controlled graphical user interfaces which aims at finding a solution to this problem by utilizing speech as input modality for graphical user interfaces. We begin with defining a taxonomy which differentiates the state of the art into two groups: speech-based mouse emulation and command-and-control. Speech-based mouse emulation uses spoken commands to emulate the functions of the mouse device (the keyboard, respectively). The user manipulates graphical objects indirectly by the functions of the mouse cursor, which are triggered by spoken commands. Command-and-control uses spoken commands to facilitate a direct control of interactive functions of graphical objects. Speech-based mouse emulation is universally applicable due to the operating system-wide available functions of the mouse cursor, whereas the spoken commands for command-andcontrol must be supported explicitly by each specific graphical object. Speech-based mouse emulation requires more user interaction than command-and-control. Thus, command-andcontrol requires less time to complete tasks than speech-based mouse emulation. However, the task completion times which can currently be achieved using command-and-control are at least 50% higher than the task completion times which can be achieved using mouse-controlled graphical user interfaces. For achieving lower task completion times with command-and-control we propose the approach conversation-and-control which is based on an extension of command-and-control by three new interaction techniques called qualified activation, automatic information error recovery and qualified feedback. Qualified activation and automatic information error recovery aim at reducing the average number of necessary spoken commands, and qualified feedback aims at reducing the average length of necessary spoken commands. Thus, we achieve a reduction of the average interaction time which results in a reduction of the average task completion time. We derive the theoretical concepts for conversation-and-control from a general model for speech-controlled graphical user interfaces, which we have developed during this work. In the first instance, we use a formal approach to verify these concepts. To show the practical feasibility of conversation-and-control, we develop an implementation concept for conversation-andcontrol on the basis of the formal model for speech-controlled graphical user interfaces. We realize the implementation concept in the form of a framework for conversation-and-control. Using this framework we conduct an experiment, which provides empirical confirmation of the formally predicted reduction of the average task completion time of conversation-andcontrol compared to command-and-control. We furthermore create prototypes of applications with graphical user interfaces supporting conversation-and-control in the application domain mobile maintenance.